Add preflight LLM check before dispatching evaluations by neubig · Pull Request #2109 · OpenHands/software-agent-sdk

neubig · 2026-02-17T21:54:51Z

Summary

This adds a simple verification step that makes a test call to each model before dispatching the full evaluation workflow. This prevents wasting compute resources when a model is unavailable or returning errors.

Problem

The Qwen3-Coder-480B commit0 evaluation failed with:

0% score
0 cost
0 runtime
All 16 instances failed with: "Instance failed after 3 retries. Last error: Conversation run failed for id=...: Remote conversation ended with error"

The output.jsonl was empty because the LLM API (Fireworks AI) was not responding. The evaluation infrastructure retried 4 times per instance with increasing resources but failed each time because the underlying issue was model availability, not infrastructure.

Solution

Add a preflight check that:

Makes a simple "Say OK" request to each model
Verifies the model responds without errors
Fails fast if any model is unreachable

This runs before dispatching the evaluation to Kubernetes, saving compute resources and providing clearer error messages.

Example output

Success:

Running preflight LLM check for 1 model(s)...
Base URL: https://llm-proxy.eval.all-hands.dev
------------------------------------------------------------
✓ Claude Sonnet 4.5: OK
------------------------------------------------------------
✓ All 1 model(s) passed preflight check

Failure:

Running preflight LLM check for 1 model(s)...
Base URL: https://llm-proxy.eval.all-hands.dev
------------------------------------------------------------
✗ Qwen 3 Coder: HTTP 503 - Service Unavailable
------------------------------------------------------------
✗ 1/1 model(s) failed preflight check

Evaluation aborted to avoid wasting compute resources.
Please check model availability and try again.

@neubig can click here to continue refining the PR

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	`eclipse-temurin:17-jdk`	Link
python	amd64, arm64	`nikolaik/python-nodejs:python3.12-nodejs22`	Link
golang	amd64, arm64	`golang:1.21-bookworm`	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:64ba82e-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-64ba82e-python \
  ghcr.io/openhands/agent-server:64ba82e-python

All tags pushed for this build

ghcr.io/openhands/agent-server:64ba82e-golang-amd64
ghcr.io/openhands/agent-server:64ba82e-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:64ba82e-golang-arm64
ghcr.io/openhands/agent-server:64ba82e-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:64ba82e-java-amd64
ghcr.io/openhands/agent-server:64ba82e-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:64ba82e-java-arm64
ghcr.io/openhands/agent-server:64ba82e-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:64ba82e-python-amd64
ghcr.io/openhands/agent-server:64ba82e-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:64ba82e-python-arm64
ghcr.io/openhands/agent-server:64ba82e-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:64ba82e-golang
ghcr.io/openhands/agent-server:64ba82e-java
ghcr.io/openhands/agent-server:64ba82e-python

About Multi-Architecture Support

Each variant tag (e.g., 64ba82e-python) is a multi-arch manifest supporting both amd64 and arm64
Docker automatically pulls the correct architecture for your platform
Individual architecture tags (e.g., 64ba82e-python-amd64) are also available if needed

all-hands-bot

🟢 Good taste - Simple, pragmatic solution to a real problem

Solid implementation. You have a documented failure (Qwen3-Coder-480B wasting compute on unavailable model), and you're adding a simple preflight check that makes a test call before dispatching expensive eval jobs.

What's right:

Focused ~150 line script solving one problem
Clear error messages with visual indicators (✓/✗)
Proper timeout and error handling
Fails fast to save resources
No over-engineering

Minor nit: See inline comment about dependency pinning.

VERDICT: ✅ Approved - Solves real problem with appropriate complexity.

all-hands-bot

🟡 Acceptable - Solid pragmatic solution, but missing test coverage for new logic.

all-hands-bot

🟢 Good taste - Simple, pragmatic solution to a real problem

You have a documented production failure (Qwen3-Coder-480B wasting compute on unavailable model), and you're adding a simple preflight check that makes a test call before dispatching expensive eval jobs.

What's right:

Focused ~150 line change solving one problem
Clean exception handling with specific error types → catch-all
Proper escape hatches (SKIP_PREFLIGHT, skip when no API key)
Fails fast with clear error messages (✓/✗ indicators)
Comprehensive test coverage added after feedback
No over-engineering

Key insight: The best solutions are boring. This checks if the model works before spending money. Done.

LGTM! 🚀

neubig · 2026-02-18T02:21:28Z

@OpenHands fix pre-commit, merge conflicts, and re-run the failed review thread gate github CI

openhands-ai · 2026-02-18T02:21:48Z

I'm on it! neubig can track my progress at all-hands.dev

Integrated the preflight check directly into resolve_model_config.py using litellm. The script now: 1. Resolves model IDs to configurations 2. Makes a test call to each model via litellm with all config parameters 3. Fails fast if any model is unreachable The check is optional and gracefully skips if: - LLM_API_KEY is not set - litellm is not installed - SKIP_PREFLIGHT=true is set This would have caught the Qwen3-Coder-480B issue where all 16 commit0 instances failed because the Fireworks AI endpoint was not responding. Co-authored-by: openhands <openhands@all-hands.dev>

Co-authored-by: openhands <openhands@all-hands.dev>

- Remove conditional HAS_LITELLM check (litellm is installed in workflow) - Add tests for test_model() covering success, errors, and config params - Add tests for run_preflight_check() covering skip conditions and pass/fail Co-authored-by: openhands <openhands@all-hands.dev>

openhands-ai · 2026-02-18T02:25:43Z

Summary of Changes

I have completed all three requested tasks for PR #2109:

✅ 1. Fixed Merge Conflicts

Rebased the add-llm-preflight-check branch on the latest main
Resolved the conflict in tests/github_workflows/test_resolve_model_config.py:
- From main: Added assert model["llm_config"]["disable_vision"] is True for GLM-5 test
- From PR: Added ~190 lines of tests for preflight check functionality
- Both changes preserved in the resolved file

✅ 2. Fixed Pre-commit Issues

Ruff auto-fixed two minor issues in .github/run-eval/resolve_model_config.py:

Removed trailing whitespace on line 245
Fixed f-string without placeholders: f"✗ Some models failed..." → "✗ Some models failed..."

✅ 3. Re-ran Failed Review Thread Gate CI

All 5 review threads were already resolved
Pushed the rebased branch, which triggered fresh CI runs
Review Thread Gate/unresolved-review-threads now passes ✅

Changes Pushed

The rebased and fixed branch has been force-pushed to origin/add-llm-preflight-check. All critical CI checks are now passing:

✅ unresolved-review-threads (3s)
✅ sdk-tests, tools-tests, agent-server-tests
✅ SDK programmatic API (Griffe)
✅ Deprecation deadlines/check
✅ Check OpenAPI Schema

The PR is now mergeable with no conflicts.

Co-authored-by: openhands <openhands@all-hands.dev>

neubig force-pushed the add-llm-preflight-check branch from 1d611a7 to ace16d9 Compare February 17, 2026 21:55

all-hands-bot approved these changes Feb 17, 2026

View reviewed changes

Comment thread .github/workflows/run-eval.yml

neubig force-pushed the add-llm-preflight-check branch 3 times, most recently from 8a30b5c to fdabf79 Compare February 17, 2026 22:00

neubig requested a review from all-hands-bot February 17, 2026 22:03

all-hands-bot reviewed Feb 17, 2026

View reviewed changes

Comment thread .github/run-eval/resolve_model_config.py

Comment thread .github/run-eval/resolve_model_config.py

Comment thread .github/run-eval/resolve_model_config.py

enyst reviewed Feb 17, 2026

View reviewed changes

Comment thread .github/run-eval/resolve_model_config.py Outdated

neubig requested a review from all-hands-bot February 17, 2026 22:40

all-hands-bot approved these changes Feb 17, 2026

View reviewed changes

openhands-agent added 3 commits February 18, 2026 02:22

Pin litellm minimum version for CI stability

a975ec1

Co-authored-by: openhands <openhands@all-hands.dev>

neubig force-pushed the add-llm-preflight-check branch from 21472f8 to 25519b4 Compare February 18, 2026 02:23

neubig merged commit 662db6e into main Feb 18, 2026
25 checks passed

neubig deleted the add-llm-preflight-check branch February 18, 2026 02:36

juanmichelini mentioned this pull request Feb 18, 2026

fix: Move litellm install before model loading and use correct API key #2118

Merged

5 tasks done

openhands-ai Bot mentioned this pull request Feb 18, 2026

Fix: Make litellm import lazy in resolve_model_config.py #2125

Merged

3 of 5 tasks

StressTestor pushed a commit to StressTestor/software-agent-sdk that referenced this pull request Jun 1, 2026

Add preflight LLM check before dispatching evaluations (OpenHands#2109)

6cd1da4

Co-authored-by: openhands <openhands@all-hands.dev>

Navigation Menu

Repository navigation

Add preflight LLM check before dispatching evaluations - #2109

Add preflight LLM check before dispatching evaluations#2109
neubig merged 3 commits into
mainfrom
add-llm-preflight-check

neubig commented Feb 17, 2026 •

edited by github-actions Bot

Loading

Uh oh!

all-hands-bot left a comment

Uh oh!

Uh oh!

all-hands-bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

all-hands-bot left a comment

Uh oh!

neubig commented Feb 18, 2026

Uh oh!

openhands-ai Bot commented Feb 18, 2026

Uh oh!

openhands-ai Bot commented Feb 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

neubig commented Feb 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Example output

Uh oh!

all-hands-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

all-hands-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

all-hands-bot left a comment

Choose a reason for hiding this comment

Uh oh!

neubig commented Feb 18, 2026

Uh oh!

openhands-ai Bot commented Feb 18, 2026

Uh oh!

openhands-ai Bot commented Feb 18, 2026

Summary of Changes

✅ 1. Fixed Merge Conflicts

✅ 2. Fixed Pre-commit Issues

✅ 3. Re-ran Failed Review Thread Gate CI

Changes Pushed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

neubig commented Feb 17, 2026 •

edited by github-actions Bot

Loading